Network-connected computing systems often utilize network packets for communications within network communication systems. Timing information related to these network packet communications is often useful so that network nodes can synchronize their own operations, synchronize their operations with other network nodes, and/or determine relative time differences between internal or external events. Timing information can be obtained using a variety of techniques. For example, timing protocols can use GPS (Global Positioning System) or other GNSS (Global Navigation Satellite System) timing signals that are determined using GPS or GNSS receivers. Time code communication formats such as TRIG (inter-range instrumentation group) can also be used to transfer timing information with respect to computing systems. Further, packet-based time synchronization techniques can also be used, such as for example, NTP (Network Time Protocol), PTP (Precision Time Protocol), CES (Circuit Emulation Service), SAToP (Structure-agnostic Time-Division-Multiplexing over Packet), and/or other packet-based timing protocols.
Certain network systems that use PTP solutions implement PTP master clocks, PTP slave clocks, PTP transparent clocks, and/or PTP boundary clocks according to the IEEE 1588 standard. In network switches, a PTP transparent clock effectively measures the time a packet takes to traverse the hardware switch. This intra-switch delay time, known as the “residence time,” can be added into a time correction field for the PTP packet in question or can be communicated in a follow-up packet in case of two-step clock synchronization communications. Downstream clocks can then account for the exact delays that packets have experienced in traversing intervening hardware switches. This residence time correction is particularly useful for synchronization using PTP packets where time is communicated by adding a precise origin timestamp associated with a master clock when the packet is transmitted. The time at the slave clock is then determined by adding the cumulative delays in correction fields to the origin timestamp contained in the PTP packet. The mechanism of the transparent clock is to perform a measurement of the intra-switch delay time required for a packet to traverse a hardware switch.
FIG. 1A (Prior Art) is a block diagram of an example embodiment for a server system 100 that communicates with an external network 118 using a timing protocol such as NTP (Network Time Protocol) and PTP (Precision Time Protocol). The server system 100 includes a central processing unit (CPU) 102 that runs an operating system 104 and one or more applications 106 that run on top of the operating system 104. The operating system 104 and the applications 106 can be implemented, for example, as computer-readable instructions stored in a non-transitory data storage medium that are accessed and executed by one or more processing devices, such as CPU 102, to perform the functions for the server system 100. An interconnect bridge 108 can also be provided to couple the CPU 102 to additional circuitry and devices within the server system 100. For example, a system oscillator 110, a real-time clock 112, a network interface card (NIC) 115, and other hardware (H/W) 114 can be coupled to the CPU 102 through the interconnect bridge 108. The system oscillator 110 can also have a direct connection to the CPU 102. The NIC 115 can also include an oscillator 116. The real-time clock 112 can be implemented using an integrated circuit including real time clock (RTC) circuitry and non-volatile read only memory (NVRAM) integrated within an integrated circuit (IC).
In operation, the hardware real-time clock (RTC) 112 provides a precise hardware timing reference for the server system 100, and accurate timing for the server system 100 is dependent upon the hardware system oscillator 110 operating at a known and consistent frequency. The operating system (OS) clock for the operating system 104 and thereby for the user applications 106 is set to this hardware RTC 112 only during system boot or start-up for the server system 100. This OS clock is subsequently maintained during operation of the server system 100 based upon ticks or sub-second time intervals counted by the CPU 102 according to the system oscillator 110. The hardware RTC 112 and the OS clock, however, can drift apart due to differences in the oscillation frequency for the system oscillator 110 and the clock frequency for the RTC 112. The operating system 102 can also be configured to set its time to an external reference using a timing protocol such as NTP or PTP. As such, the OS clock is then synchronized periodically to a NTP/PTP reference time over the external network 118. The OS clock is maintained between NTP/PTP updates (e.g., one second or more intervals) based again upon ticks or sub-second time intervals counted by the CPU 102 according to the system oscillator 110. The trigger within the server system 100 to obtain NTP/PTP updates is based on duration of interval time as determined using the OS clock. As such, time drifts can occur between reference time and OS time due to inaccuracy of system oscillator 110 used to count time between NTP/PTP updates.
FIG. 1B (Prior Art) is a process flow diagram of an example embodiment 150 for timing associated with embodiment of FIG. 1A (Prior Art). As described above, the real-time clock (RTC) 112 can be implemented using an integrated circuit including real time clock (RTC) circuitry and non-volatile read only memory (NVRAM). Further, as indicated by block 160, this real-time clock 112 can be configured to keep time continuously and can be powered, for example, by a battery included within a chassis for the server system 100. This real time clock (RTC) 112 is then used to initialize the system time. In block 152, the server system 100 powers up, and the operating system software 104 loads in block 154. In block 156, the system time is set according to the real-time clock (RTC) 112. In addition, the operating system has the CPU count clock ticks based upon the system oscillator 110 to maintain system time, as indicated by block 162 and the arrow from block 162 to block 156. The server system 100 finishes the boot process in block 158. During operation, a determination is also made in block 164 whether NTP/PTP is being used to update system time within the server system 100 to an external network timing source. If “NO,” the flow passes back to block 162 where system time is maintained simply by counting ticks or sub-second time intervals of the system oscillator 110. If “YES,” then flow passes to block 166 where it determined whether an NTP/PTP update is required. If “NO,” then flow again passes back to block 162. If “YES,” then flow passes to block 168 where the operating system (OS) clock or system time is refreshed using NTP/PTP to obtain a reference time from an external timing source through the external network 118.
Certain network communication systems include virtualized processing environments, such as virtual machine (VM) platforms hosted by one or more processing devices, to provide processing resources to user processing systems. For example, network cloud resources made available to network-connected systems are often virtualized such that processors or processing cores associated with a server processing platform (e.g., server blade) and/or combinations of such server processing platforms are used to provide software processing instances or virtual machine platforms within cloud server processing systems. A virtual machine (VM) platform is an emulation of a particular processing system that is created within software being executed on a VM host hardware system. By creating VM platforms within a VM host hardware system, the processing resources of that VM host hardware system can be more easily shared among network connected systems that desire to use these processing resources.
FIG. 2A (Prior Art) is a block diagram of an example embodiment for a VM host hardware system 200 that communicates with an external network 118 using a timing protocol such as NTP or PTP. The VM host hardware system 200 can be implemented in a similar fashion as the server system 100 of FIG. 1A (Prior Art) with respect to the CPU 102, the interconnect bridge 108, the RTC 112, other hardware (H/W) 114, and the NIC 115. Further, the VM host hardware system 200 also includes VM host operating system 204 that in part can operate in a similar fashion to the operating system 104 for embodiment 100 of FIG. 1A (Prior Art) with respect to system clock timing.
In contrast with embodiment 100, however, the VM host hardware system 200 includes a hypervisor 205 that executes on top of the VM host operating system (OS) 204. This hypervisor 205 provides a virtualization layer including a plurality of VM platforms 206A, 206B, 206C . . . that emulate processing systems and related processing resources, such as the server system 100 depicted in FIG. 1A (Prior Art). As shown with respect to VM platform 206A, therefore, each of the VM platforms 206A, 206B, and 206C have one or more virtual hardware resources associated with it, such as a virtualized network interface card (NIC) 208A, a virtualized CPU 210A, a virtualized real-time clock (RTC) 212A, and/or other virtualized hardware resources. The VM host hardware system 200 makes each of the VM platforms 206A-C available for use by one or more network-connected guest systems through the VM host operating system 204 and the hypervisor 205. It is noted that the hypervisor 205 provides a management and control virtualization interface layer between the VM platforms 206A-C and the guest systems using the processing resources provided by the VM platforms 206A-C. It is further noted that the VM host operating system 204, the hypervisor 204, the VM guests 206A-C, and the virtualized hardware resources 208A/210A/212A can be implemented, for example, as computer-readable instructions stored in a non-transitory data storage medium that are accessed and executed by one or more processing devices, such as the CPU 102, to perform the functions for the VM host hardware platform 200.
In operation, the VM platforms 206A-C rely upon virtualized hardware clocks for timing even though these clocks are virtualized and not actually present. The virtualized CPU 208A, for example, does not operate at consistent speed as its clock speed may vary according to processing load on the software instance providing the VM platform 206A. Further, clock ticks for the virtualized CPU 208A typically have significant variations leading to large timing errors. In addition, NTP/PTP operations for the VM platform 206A will be set to use the operating system time for the VM host hardware system 200 for NTP/PTP updates. As such, large time errors (e.g., 1 second or more) can accumulate between NTP/PTP synchronization update intervals, because the stimulus to update using NTP/PTP is based on counting system oscillator ticks, and these ticks can be slowed dramatically with respect to the VM platform 206A. Because NTP/PTP depends on symmetric and consistent network delays, the high variability for the VM platforms 206A-C reduce accuracy for NTP/PTP. The virtual machine environment, therefore, creates significant inaccuracies with respect to timing synchronization.
FIG. 2B (Prior Art) is a process flow diagram of an example embodiment 250 for timing associated with embodiment of FIG. 2A (Prior Art). As described above, the VM host operating system 204 and hypervisor 205 emulates a virtualized real-time clocks (RTC) 212A based upon the host OS system time along with the VM platform 206A. As indicated by block 260, this virtualized real-time clock 212A is used to initialize the system time. In block 252, the VM platform 206A powers up, and virtualized operating system software loads in block 254. In block 256, the system time for the VM platform 206A is set according to the virtualized real-time clock (RTC) 212A. In addition, the operating system has the virtualized CPU 210A count clock ticks based upon a virtualized system oscillator to maintain system time, as indicated by block 262 and the arrow from block 262 to block 256. The VM platform 206A finishes the boot process in block 258. During operation, a determination is also made in block 264 whether NTP/PTP is being used to update system time within the VM platform 206A. If “NO,” the flow passes back to block 262. If “YES,” then flow passes to block 266 where it determined whether an NTP/PTP update is required. If “NO,” then flow again passes back to block 262. If “YES,” then flow passes to block 268 where the virtual operating system (OS) clock or system time is refreshed using NTP/PTP. However, the reference clock for this NTP/PTP update will by the system time for the VM host hardware system 200. As described above, significant errors (e.g., 1 second and above) can be introduced by variations within processing loads for the VM platform 206A between these timing updates and based upon timing errors within the VM host hardware system 200.
FIG. 3A (Prior Art) is a block diagram of an embodiment 300 where a virtual switch 304 has been included along with the hypervisor 205 within a virtualization layer 302. The virtualization layer 302 can provide one or more virtualized hardware components, such as for example virtualized input/output (IO) interfaces, virtualized network interfaces, virtualized CPUs, virtualized storage mediums, and/or other virtualized components. The virtual switch 304 provides virtual network packet forwarding among the VM platforms 206A, 206B, 206C, and 206D that are hosted as guest processes within the host operating system 204. In particular, the virtual switch 304 communicates with the virtualized NICs 208A, 208B, 208C, and 208D for the VM guest platforms 206A, 206B, 206C, and 206D to forward network packets among the VM guest platforms 206A, 206B, 206C, and 206D and between the VM guest platforms 206A, 206B, 206C, and 206D and the external network 118. As an element of the virtualization layer 302, the virtual switch 304 operates using system clock information from the system oscillator 110, as represented by the dashed line between them.
Although the virtual switch 304 expands the applications that can be effectively hosted by the VM host hardware system 300, the operation of the virtual switch 304 causes variable unknown delays for packets traversing the virtual switch 304, as represented by block 306. These variable unknown delays create errors in timing synchronization that cause the VM host hardware system 300 to be unacceptable for timing sensitive applications.
FIG. 3B (Prior Art) is a message diagram of an embodiment 350 for master/slave messages for a timing protocol such as NTP or PTP. The timing messages allow the slave clock 370 to align itself with the master clock 360. Packet exchanges between master 360 and slave 370 provide measurements of the transit delay between the two using timestamps communicated between the two entities. Embodiment 350 provides an example sequence of events and timing messages associated with an exchange of master and slave timing packets between master 360 and slave 370. The events depicted are:                Event A 330: Packet (in PTP this is the Sync_message) is transmitted by master 360 and the time-of-departure 351 is t1.        Event B 332: The packet (Sync_message) arrives at slave 370 that measures the time-of-arrival as τ2; assuming that the slave time offset from master (ofm) is ε, the actual time-of-arrival 352 is t2=τ2+ε.        Event C 334: Packet (e.g., the PTP message referred to as Delay_request) is transmitted by slave 370 that notes the time-of-departure is τ3; assuming that the slave time offset from master is ε, the actual time-of-departure 353 is t3=τ3+ε.        Event D 336: The packet (Delay_request) arrives at master 360 that measures time-of-arrival 354 as t4.        Event E 337: Packet (the Delay_response message) is sent from the master 360 towards the slave 370, and the content of the packet contains the time-of-arrival t4 354 (of the Delay_request at the master).        Event F 338: The packet (Delay_response) arrives at the slave 320. Now the slave clock 370 has four time-of-arrival and time-of-departure values associated with this exchange of packets and these four values allow for time synchronization of the slave clock to the master clock.        
Such a two-way exchange of packets can provide information suitable for allowing the slave 370 to align in time with the master 360. It is also noted that one-way exchanges of timing information can also be used. For one-way communications from master 360 to slave 370, the slave 360 can align its clock with the master 370 if the packet contains the time-of-departure 351 from the master (t1) and the slave 370 measures the time-of-arrival (τ2). For one-way communication from the slave 360 to the master 370, the slave 360 can align its clock with the master 370 if a mechanism is available for the slave 370 to obtain the results of measurements at the master 370 of time-of-arrival at the master (t4).
For two-way exchanges as shown in embodiment 350, there are four measured values that are communicated between the master 360 and slave 370, namely, (t1, τ2, τ3, t4). It is noted that such a two-way exchange involves one timing message packet in each direction, and these timing message packets do not have to be consecutive as long as the time-stamp information is communicated appropriately. In some instances, the rate at which timing packets are transmitted in the two directions can be different. Denoting by ΔMS 340 and ΔSM 344 the transit delays between the master 360 and slave 370 and vice versa, the following equations can be established:t4=τ3+ε+ΔSM (from Slave to Master packet)t1=τ2+ε−ΔMs (from Master to Slave packet)  (Eq. 1)As there are two equations with three unknowns, it is common to assume reciprocity of transit delay between the two devices, thereby reducing the number of unknowns to two and therefore computing the slave time offset (ε) from master from (Eq. 2).
                    ɛ        =                                                            (                                                      t                    4                                    +                                      t                    1                                                  )                            -                              (                                                      τ                    3                                    +                                      τ                    2                                                  )                                      2                    =                                                    (                                                      t                    4                                    -                                      τ                    3                                                  )                            -                              (                                                      τ                    2                                    -                                      t                    1                                                  )                                      2                                              (                  Eq          .                                          ⁢          2                )            
In a virtual environment such as for the VM guest platforms 206A-D in FIG. 3A, however, the transit delays are highly variable due to the operation of the virtualization layer 302 and the virtual switch 304. As such, the estimate of the slave time offset (ε) determined by one of the VM guest platforms 206A-D trying to synchronize to a PTP master clock within the external network 118 will potentially have significant time errors from packet to packet depending upon variations within the time-of-transit for each packet through the virtual switch 304.